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Abstract. The purpose of the present paper is to discuss the following conjecture of 
Fel'shtyn and Hill, which is a generalization of the classical Burnside theorem: 

Let G be a countable discrete group, its automorphism, -R((/)) the number of (jy 
conjugacy classes (Reidemeister number), S{4)) = #Fix(0) the number of 0-invariant 
equivalence classes of irreducible unitary representations. If one of R{(j)) and S{4i) is 
finite, then it is equal to the other. 

This conjecture plays a very important role in the theory of twisted conjugacy classes 
having a long history (see [14], [4]) and has very serious consequences in Dynamics, 
while its proof needs rather fine results from Functional and Non-commutative Harmonic 
Analysis. It was proved for finitely generated groups of type I in [10]. 

In the present paper this conjecture is disproved for non-type I groups. More precisely, 
an example of a group and its automorphism is constructed such that the number of fixed 
irreducible representations is greater than the Reidemeister number. But the number of 
fixed finite-dimensional representations (i.e. the number of invariant finite-dimensional 
characters) in this example coincides with the Reidemeister number. 

The directions for search of an appropriate formulation are indicated (another defini- 
tion of the dual object). 
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1. Introduction: motivation, history, and current state 

Definition 1.1. Let G be a countable discrete group and : G ^ G an endomorphism. 
Two elements x^x' E G are said to be (p-conjugate or twisted conjugate, iff there exists 
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g E G with 

x' = gx(f){g~'^). 

We shall write {x}^ for the (l)-conjugacy or twisted conjugacy class of the element x & G. 
The number of 0-conjugacy classes is called the Reidemeister number of an endomorphism 
(f) and is denoted by R{(f)). If is the identity map then the 0-conjugacy classes are the 
usual conjugacy classes in the group G. 

If G is a finite group, then the classical Burnside theorem (see e.g. [15, p. 140]) says that 
the number of classes of irreducible representations is equal to the number of conjugacy 
classes of elements of G. Let G be the unitary dual of G, i.e. the set of equivalence classes 
of unitary irreducible representations of G. 

Therefore, by the Burnside's theorem, if is the identity automorphism of any finite 
group G, then we have = #Fix(0), where 0[p] = [p o 0]. 

One of the main achievements in the field till now is the following result. 

Theorem 1.2 ([10]). Let G he a finitely generated discrete group of type 1, one of its 
endomorphism, R{(j)) the number of (p- conjugacy classes, and S{(f)) = 7^Fix(0) the number 
of (p-invariant equivalence classes of irreducible unitary representations. If one of R{(p) 
and S{(f)) is finite, then it is equal to the other. 

The research is motivated not only by a natural desire to extend the classical Burnside 
theorem to the case of infinite groups and twisted conjugacy classes, but also by dynamical 
applications. Namely, the identification of Reidemeister number with a number of fixed 
points in a natural way, has some very interesting consequences. 

More precisely, let n{d), d E N, he the Mobius function, i.e. 

( 1 if rf= 1, 
jj,{d) = < (—1)^ if (i is a product of k distinct primes, 
[^0 if d is not square — free. 

Theorem 1.3 (Congruences for the Reidemeister numbers [10]). Let (p : G ^ G be an 
endomorphism of a countable discrete group G such that all numbers R{(j)"') are finite and 
let H be a subgroup of G with the properties 

(f>{H) C H, 

Vx e G 3n e N such that (t/\x) e H. 

If for the pair {II,(j)'^) the twisted Burnside theorem holds, i.e. this pair satisfies the 
conclusion of Theorem 1.2 for any n e N, then one has for all n, 

^p(rf) ■i?(0"/'^) = modn. 

d\n 

These theorems were proved previously in a special case of a direct sum of an Abelian 
finitely generated group and finite group [6, 7]. Remember that the difference between 
result of [6, 7] and Theorem 1.2 is not so big because by theorem of E.Thoma a countable 
finitely generated group G is a group of type I iff it is a finite extension of an Abelian 
group. The conjecture under discussion was formulated in [6]. We refer to [4, 10] for more 
detail and topological applications. 
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On the other hand, one can introduce the number of " Reidemeister classes related 

to twisted invariant functions on G from the Fourier- Stieltjes algebra B{G)" , or more 
precisely, the dimension of the space of twisted invariant functions on G which can be 
extended up to bounded functionals on the group algebra C*{G). Let S^{(j)) be the sum of 
codimensions of subspaces Lj C C*{G)/I, where Lj is generated by elements of the form 
a — LgaL^(^g-i) and / runs over the Glimm spectrum T of G. Let us remind this notion. 

Let Z be the center of a C*-algebra A and Z its space of maximal ideals equipped with 
the standard topology. If / G P := Prim(A) (the space of kernels of unitary irreducible 
representations), then Z (1 1 E Z (this follows from the fact that the restriction onto Z 
of an irreducible representation with kernel / gives rise to a homomorphism Z — C, and 
hence is a maximal ideal Z (1 1). We obtain a map f : V ^ Z. Suppose T :— f{V). For 
each X eT consider the ideal Ix :— n/, /(/) = x, {Glimm ideal) and the field of algebras 
A/Lj.. We have the map {x ^ a + Jj.} from the algebra A to the algebra of sections of 
the mentioned field. An important result of [3] asserts that this map is an isomorphism. 
The map / : P — > T is universal with respect to continuous maps g : V ^ S to Hausdorff 
spaces, i.e. any such map can be represented under the form /i o / for some continuous 
h : T ^ S. The space T is compact for a unital algebra. 

We call S^{(f)) the number of generalized fixed points of on the Glimm spectrum of 

G. 

Theorem 1.4 (weak twisted Burnsidc theorem, [19]). The number R^{(j)) is equal to the 
number S^{(f)) of generalized fixed points of (f) on the Glimm spectrum ofG, if one of R^{(j)) 
and S^{(f)) is finite. 

This result allows to obtain the strong form of twisted Burnside theorem i?(0) = 'S'(0) 
in a number of cases. 

The interest in twisted conjugacy relations has its origins, in particular, in the Nielsen- 
Reidemeistcr fixed point theory (see, e.g. [14, 4]), in Selberg theory (see, eg. [18, 1]), and 
Algebraic Geometry (see, e.g. [13]). 

The congruences give some necessary conditions for the realization problem for Reide- 
meister numbers in topological dynamics. 

Let us remark that it is known that the Reidemeister number of an endomorphism of 
a finitely generated Abelian group is finite iff 1 is not in the spectrum of the restriction 
of this endomorphism to the free part of the group (see, e.g. [14]). The Reidemeister 
number is infinite for any automorphism of a non-elementary Gromov hyperbolic group 
[11, 16] as well as for any injective endomorphism of Baumslag-Solitar group [5] (see also 
[8]). 

The main result of the present paper is the following statement. 

Theorem 1.5. There exists an amenable solvable group G not of type I and its automor- 
phism (p such that its Reidemeister number R{(j)) is finite but does not coincide with the 
number S{(j)) of fixed points of cj) onG. 

This example is very important for the further attack onto the problem, because this 
group is "situated between" the groups of type I, for which the conjecture is true [10], 
and Gromov hyperbolic groups, for which Reidemeister numbers are always infinite. 
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The ideas for the further study arising from the example from this paper and from [19] 
are discussed in the last section below. 



2. Some technical preliminaries 

Let G be a semidirect product of 1? and Z by Anosov automorphism a with the matrix 
2 1 , 

(of course, our results remain valid for any hyperbolic element of SL{2, Z) 



1 1 

to use as A). It consists by the definition of triples ((m, k),n) of integers with the following 
multiplication low: 

((m, k),n) * ((m', k'),n') = ((m, k) + Q;"(m', k'),n + n'). 

In particular, 

((m,A;),0)*((0,0),n) = ((m,A;),n). 
Elements of G can be written also as matrices 



1 



n e Z, X = (m, k) e 1?. 



The inverse of ((m, k),n) is (— Q;~"^(m, /c), — n). Indeed, 

((m, k),n) * {-a~"^{m, k), -n) = ((m, k) - a'^a~"'{m, k),n- n) = ((0, 0), 0). 

The group G is a solvable (hence, amenable) group which is not of type I. Its regular 
representation is factorial. 

Let us define an automorphism (p : G ^ G hy 



0((m, k),n) = {{k, -m), -n), 

i.e. the action on Z^ is defined by automorphism /j. with the matrix M — 

and on Z by n H- > —n. The map is clearly a bijection. 



1 
-1 



0(((m, k),n)* ((m', k'),n') = 0((m, A;) + Q;"(m', A;'), n + n')) 

= ((/c, — m) + ^a^{m\ k'), —n — n'), 

(t>{{m, k),n) * 0(((m', k'),n') — {{k, — m), — n) * ((A;', — m'), — n') 

= ((A;, -m) + q;-"(A;', -m'), -n - n'). 

Hence, to prove that (j) is an automorphism, we need /ia" = Q;^"/i. This follows from 
//a = a~^ii. The further results in this direction can be found in [12]. 

One can hope to find 0-fixed irreducible representations (at least finite-dimensional 
ones) from those orbits of a on the dual torus to the normal subgroup Z^, which are 
invariant under the dual action of ii, using appropriate cocycles. This will be done in the 
last section. We will find four such representations and we will explain why that is all. 
Some motivation for this way of construction can be found in [2, Ch. 17, § 1]. 
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3. Description of Reidemeister classes 

Let us find the Reidemeister classes of 4>, i.e. the classes of the equivalence relation 
h ~ gh(f){g~^). We will prove the following statement. 

Lemma 3.1. For the above G and (j) one has R{(t)) = 4. 

Also we will obtain a description of these four Reidemeister classes. 
For h — {{m,k),n) and g = {{x,y),z) the right hand side of the relation takes the 
following form: 

((x, y) + ot^im, k),z + n) * {—^a~^{x, y),z) = 
{{x, y) + Q;^(m, k) — a^~^"'iia~^{x, y),2z + n) — 

(1) = {a''{{m, k) + (Id -a»a"'=(a;, y)}, 2z + n). 

Let us call level n (of G) the coset L„ of C G of all elements of the form ((m, k),n). 
Let us first take an element {{m,k).0) from the level and describe elements from the 
same level, being equivalent to it. By (1) in this case z = n = and they have the form 

{{m,k) + {Id -ii){x,y),0) = {{m + {x-y),k + {x + y)),0), 

where Id —/i has the matrix ^ j 1^ ) Hence, the level has intersections with 2 

Reidemeister classes, say, Bi and B2. The first intersection BiOLq is formed by elements 
((■u, v), 0) with odd u + v, and S2 H Lq — with even u + v. The elements from the other 
levels, which are equivalent to ((m, k),0), have the form 

(2) (a^{(m, k) + (Id -fi)a-'{x, y)}, 2z). 

This means that Bi and B2 enter only even levels. Also, since a is an automorphism, we 
can rewrite (2) as 

(3) {a'{{m,k) + {ld-i2){u,v)},2z). 

with arbitrary integers u and v. This means, that the intersections Bir\L2z have the form 
Q;^(i?i), i = 1,2. In particular, the other Reidemeister classes do not enter even levels. 
In a similar way, the elements of Li equivalent to ((m. A;), 1) have the form 

{{m,k) + {ld-aii){x,y),l) = {{m + {2x - 2y),k + x),l). 

This means, that Li enters 2 classes: S3 is formed by elements with even first coordinate, 
and B4 — with odd one. The elements from the other levels, which are equivalent to 
((m, k),l), have the form 

(4) {a'{(m, k) + (Id -aii)a-'(x, y)}, 2z + l). 
Since a is an automorphism, we can rewrite (4) as 

(5) {a'{{m, k) + (Id -afj,){u, v)}, 2z + l). 

with arbitrary integers u and v. This means, that the intersections Bi fl L22+1 have the 
form a^{Bi), i = 3,4. In particular, these four classes cover G and R{(f)) = 4. 

To obtain a complete description of B^ let us remark that directly from the definition 
of a 

Oi{x,y) = {2x + y,x + y) 
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one has the following properties: 

• a maps the set of elements with an even (resp., odd) sum of coordinates onto the 
set of elements with an even (resp., odd) second coordinate, 

• a maps the set of elements with an even (resp., odd) second coordinate onto the 
set of elements with an even (resp., odd) first coordinate, 

• a maps the set of elements with an even (resp., odd) first coordinate onto the set 
of elements with an even (resp., odd) sum of coordinates. 

Hence, the eleiiieiits {{in. k). ri) in iiiterseetions Z?, H Lj are of tlie form 



i 


1 


2 


3 


4 


j = mod 6 


m + k is even 


m + k is odd 








j = 1 mod 6 








m is even 


m is odd 


j = 2 mod 6 


k is even 


k is odd 








j = S mod 6 








m + A; is even 


m + k is odd 


J = 4 mod 6 


m is even 


m is odd 








j = 5 mod 6 








k is even 


k is odd 



4. Fixed points and twisted invariant functionals 

Now we want to study the fixed points of the homeomorphism : G — > G, [p] 
of the unitary dual. We will prove the following statement. 

Lemma 4.1. There are exactly four finite- dimensional representations, which are fixed 
points of (f). There is at least one infinite- dimensional representation, which is a fixed 
point of(f). 

Let us start from the finite-dimensional representations. As it was shown in [10] in 
this case there exists exactly one twisted- invariant functional on L^{G), or 0-central 
function, coming from a twisted-invariant functional on p{L^{G)) = M(dimp, C) (up to 
scaling), namely 

(6) Vp- 9^ Tr(5p(^)), 

where S is the intertwining operator between p and p4>. This is an appropriate notion of 
character of a 0-invariant representation. 

First, we have to find //-invariant finite a-orbits on T^. One can notice that 

det(^'^-M)=det( 



ad-b^ + 1 = det A' 



where A^' 



b 
d 



, for any n. Hence, the mentioned orbits are formed by points 

with coordinates and 1/2. We have 2 orbits: one of them consists of 1 point (0,0) 
and gives rise to 1-dimensional trivial representation pi, and the other consists of Ai — 
(0, 1/2), A2 — (1/2, 0) and A^ = (1/2, 1/2) and gives rise to a 3-dimensional (irreducible) 
representation p2- Also, one has the following 1-dimensional representation tt: 

7r((m, A;),2n) = 1, 7r((m,k),2n + 1) ^ -1. 

So, we have 4 representations 

Pi, P2, TT, p2<8)7r. 
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These representations are irreducible finite dimensional (for the last one this follows from 
the evident observation (p2 ® T!'){g) = ^P2{g), hence the space of commuting operators 
is still isomorphic to C). We claim that these representations give rise via (6) to 4 
linear independent twisted-invariant functionals. In particular, there is no more finite- 
dimensional fixed points of (p. More precisely, this follows from the general theory of [10] , 
since these finite-dimensional representations are well separated. But we will calculate 
directly for the self-containing of the presentation here, and also because we need the 
explicit form of these functionals. 
Clearly, 



(7) 



1, on Un L2n = BiU B2, _ 

-1, on U„ Un+i = 5,3 U B,, "^^'^^ - "^"^ ■ 



Let us find ipp^. In the space L'^{Ai, A21 A^) we take the base £1, £2, £3 of characteristic 
functions in these points. One has 

oi{ei) = £3, a{e^) = £2, a{e2) = £1, 

A«(£i) = £2, A*(£2) = £1, IJ-isz) = £3- 
The representation (see [2, Ch. 17, § 1]) is defined by: 

P2(m,/c, 0)(£i) = XAi(?7i, fc) •£i, p2(0,0,n)(£i) = Q;~"(£i) = £j+„ mod 3, 
P2(m, k, 0)(£i) = e^-^^o-^+i/^-'^) . £1 = e"'=£i, 
P2(m, k, 0)(£2) = e2"(i/2""+°-^) • £2 = e'^^"^£2, 
P2(m, k, 0)(£3) = e^-ay-^-m+y^.k) . _ e'^^("^+^)£3. 
The representation (f)p2 is defined by 

0P2(O,O,n)(£i) = P2(0,0, -n)(£i) = £j-n mod 3, 

Jp2(m, k, 0)(£i) = p2(A:, -m, 0)(£i) = e'^'^^e^ = e'^^"^£i, 
0P2(m, A;, 0)(£2) = P2(/^, -m, 0)(£2) = e^^'=£2, 
0P2(m, A), 0)(£3) = p2(A;, -m, 0)(£3) = e-^('=-'")£3 = e^^^^+^^^es- 



The intertwining operator is induced by and has the matrix 5" 



Tr 



(Pp.,{m,k,n) = Ti{Sp2{m,k,0)p2{0,0,n)) 




Hence, 
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For n = mod 3 

(Pp^{m,k,n) = Tr 






gTTim 





gTrife 

















— e 



7ri(A;+m) 



for n = 1 mod 3 

(Pp^{m,k,n) = Tr 




1, 


if m 


-1, 


if m 












n 
u 


n 
u 









y 























gTrim 
















1, if m is even 
-1, if m is odd 






1 













1 












-TTlfe 



1, 
-1, 



if A; is even 
if k is odd 



(^P2 is 3-periodical in n, while the characteristic functions of Bi are 6-periodical. For 
j = 0, . . . ,5 one has 



fp2 


BiHLo 


= 1, 


'^P2 


B2nLo = 


-1, 


<^P2 




= 1, 


<^P2 


B4,nLi — 


-1, 




Bif\L2 




^P2 


B2f\L2 


-1, 








<^P2 


B4nL3 = 


-1, 


Vp2 




= 1, 


Vp2 


n, ! : = 


-1, 


'PP2 




= 1, 


'PP2 


B4nL6 = 


-1, 



so v^palsiuBs = 1, ¥'p2|b2UB4 = — 1. the determinant of the values of the functions (pp^^, (p-rr, 
Vp2: Vp2®T^ B2, -B3, B4 is 

1 \ 

-1 

-1 
1 



det 



/ 1 

1 
1 
1 



-8 7^ 0. 



Hence, they are linearly independent. 

Nevertheless, there arc infinite-dimensional irreducible 0-invariant representations. E.g. 
we have a representation p of G on L^(T^) with the respect to the Lebesgue measure, with 
p{m, k, 0) be the multiplier by characters in the appropriate points and p(0, 0, 1) is a, in 
the same manner as for p2- This representation can be also obtained by inducing from the 
trivial representation of the subgroup {«"} C G and Fourier transform. The irreducibility 
of this induced representation can also be proved using [17]. 

This disproves the conjecture of Fel'shtyn and Hill [6], who supposed that the Reide- 
meister number equals to the number of fixed points of on G 



AN EXAMPLE OF TWISTED BURNSIDE THEOREM 



9 



This representation is not traceable, but one can nevertheless try to calculate (6). We 
will do this in some fixed base, because in general operators are not of trace class. Let us 
choose an orthonormal base of L^(T^) formed by est{x,y) — e^'^*(*^+*2/) ^ x,y & [0, 1]. The 
intertwining operator S is generated by /x. Then 

Tr{Sp{m,k,n)) = / {p{m, k,0)p{0,0,n)es,t){fJ^{x,y))e7;-t{x,y) dx dy = 

-1 rl 



s,t -^0 -^0 

s,t 

^ /"^ /"\27ri((m,fc)+(a"-/.)(5,t),;u(a;,2/)> ^^^^ ^ 1. 



s,t (m,fe)=(M-a")(s,t) 

For n — Q this equals 1 for m + /c even and for m + odd. Hence, (^plsinio — 
'^p\B2nLQ — 0. For n — 1 the equality takes the form (m, k) — (—2s — 2t, —t). Hence, 
(/^pIsanLi = 1, Vp\BAnLi =0. If n = 2r, the equality takes form 

(m, k) — {/I — a'^^){s, t) , (m, k) — {a^/ia'^ — a^'') (s, t) , 

a~'^{m, k) = {/J, — a^)a^{s, t). 

Since a is an automorphism of Z © Z and by the description of Bi via the action of a, we 
obtain that (pp\Bi = 1, ^p\b2 — 0- Similarly, for n odd. So, (pp is well defined and 

Vp = XB^ + XB:i = ^{fp, + 'Pp2)- 

Of course, there are also </)-invariant traceable factor representations of this group G, 
e.g. the regular representation. Since its kernel is trivial, evidently all twisted-invariant 
functionals can be pulled back from it. 



5. Discussion, prospectives, conjectures 

This example shows that the conjecture of [6] for general groups can survive only 
after eliminating badly separated points in G. A partial solution of the problem along 
this direction is obtained in Theorem 1.4. As it is known, the unitary dual of a group 
of polynomial growth is well separated, this gives hope to get an analytical proof of a 
version of twisted Burnside theorem for these groups. Also, while the present paper was 
under refereeing A.F. and E.T. have obtained the following general theorem: for any 
automorphism of any almost polycyclic group R{4>) = >S'/(0) if one of them is finite, where 
Sf{(f)) is the number of fixed points of on the finite-dimensional part of G ([9], see also 
the survey [8] for a number of related statements, counterexamples, etc.). 
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